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The quintessence of intelligence is one of the big questions still beyond our understanding. 
In the past, science has unravelled many other previously puzzling questions through 
measurement, a fundamental tool for the identification, comparison and classification of 
natural phenomena. Not surprisingly, a very significant portion of our still scant knowledge 
about what intelligence is - and what it is not - comes from this measurement effort. For 
more than a century, psychometrics, comparative psychology and other disciplines have 
developed a rich collection of measurement instruments for quantifying various behavioural 
properties in the animal kingdom, prominently placing humans as a yardstick. 

Beyond the enormous landscape of behaviours in the animal kingdom, there is yet another 
gigantic space to be explored: the machine kingdom. A plethora of new types of'creatures’ is 
emerging: robots, animats, chatbots, digital assistants, social hots, automated avatars and 
artificial life forms, to name a few, including hybrids and collectives, such as machine- 
enhanced humans, cyborgs, artificial swarms, human computation systems and crowd 
computing platforms. These systems display behaviours and capabilities as peculiar as their 
developers and constituents can contrive. Universal psychometrics presents itself as a new 
area dealing with the measurement of behavioural features in the machine kingdom, which 
comprises any interactive system, biological, artificial or hybrid, individual or collective. 

The focus on an enlarged set of subjects generates plenty of new questions and 
opportunities. Are IQ tests valid for arbitrary machines? Can we devise universal cognitive 
tests? Can we have a formal definition of intelligence solely based on computational 
principles? Can the structure of cognitive abilities and empirical latent factors, including the 
dominant g factor, be extrapolated beyond biological creatures? Can this be studied 
theoretically? How should artificial personalities be measured? Do we need intelligence to 
evaluate intelligence universally? The classical paradigms used to evaluate natural and 
artificial systems have not been able to answer (or even formulate) these questions 
precisely. Also, customary evaluation tools are gamed by these new kinds of systems. 

Recently, however, there has been a significant progress in a principled approach to the 
evaluation of behaviour based on information theory and computation. The anthropocentric 
stance is replaced by a universal perspective where life forms are considered as particular 
cases. Classical tools in human psychometrics, comparative psychology and animal cognition 
are not jettisoned but rethought for a wider landscape and substantiated on algorithmic 
grounds. 

This book provides a comprehensive account of the concepts, terminology, theory and 
tools that should compose a unified framework for the universal evaluation of behavioural 
features. The exposition does not avoid some notions that are less consolidated, such as the 
arrangement of the space of abilities, the evaluation of personality or the process of ability 
development. The ideas that do not work are openly criticised, to aid the understanding of 
the many scattered scientific contributions that have recently appeared in different areas. In 
fact, some of these theories only make real sense - or no sense at all - when they are put 
together. 



Many of the current conundrums in the evaluation of natural intelligence derive from the 
empirical evaluation of 'populations’ (human groups, age ranges, species, etc.). The 
consideration of any conceivable behaviour (natural or artificial) and any imaginable 
'machine population’ provides a falsifiability criterion for any general claim, theory or test 
about behavioural features. The machine kingdom also brings a myriad of subjects to 
evaluate, with fewer experimentation constraints than those posed by humans and other 
animals. The theoretical underpinning on computation and information theory leads to 
several key formalisations, such as the concepts of task difficulty and policy-general 
intelligence. These new grounds illuminate blatant questions such as what human 
intelligence tests really measure. 

Artificial intelligence can also benefit from the distinction between task-oriented 
evaluation and feature-oriented evaluation, jointly with a less anthropocentric methodology 
for the development and assessment of general-purpose agents. If properly overhauled, 
many tools from psychometrics can enter the scene of artificial intelligence evaluation, such 
as item response theory and adaptive testing. Similarly, the experience in the design of 
interfaces from animal evaluation can be crucial beyond natural intelligence. 

Psychometrics, comparative psychology and artificial intelligence evaluation usually 
speak different languages. A great effort has been made to render this book accessible and 
valuable for researchers and students in all these areas and, extensively, to any interested 
reader outside these disciplines. As a result of the integration of different areas, some 
paradigms will be challenged and some hypotheses will be refuted. The outcome for the 
future is an integration of well-founded principles for the evaluation of behaviour in humans, 
non-human animals and all other machines. 
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